Image-capturing apparatus and method, expression evaluation apparatus, and program

ABSTRACT

An image-capturing apparatus for capturing an image by using a solid-state image-capturing device may include a face detector configured to detect a face of a human being on the basis of an image signal in a period until an image signal obtained by image capturing is recorded on a recording medium; an expression evaluation section configured to evaluate the expression of the detected face and to compute an expression evaluation value indicating the degree to which the detected face is close to a specific expression in relation to expressions other than the specific expression; and a notification section configured to notify notification information corresponding to the computed expression evaluation value to an image-captured person.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.11/881,989, filed on Jul. 30, 2007, which claims priority from JapanesePatent Application No. JP 2006-211000 filed in the Japanese PatentOffice on Aug. 2, 2006, the entire content of which is incorporatedherein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image-capturing apparatus and methodfor capturing an image by using a solid-state image-capturing device, toan expression evaluation apparatus for evaluating the expression of animage-captured face, and to a program for performing the processingthereof.

2. Description of the Related Art

In image-capturing apparatuses, a so-called self-timer function ofautomatically releasing a shutter button after a fixed period of timehas elapsed from when an operation for depressing the shutter isperformed is typically installed in not only silver-halide cameras, butalso digital still cameras. However, the timing at which a shutter isreleased by the self-timer function is determined in advance. Therefore,it is difficult to guarantee that a person being image-captured has adesired expression on their face at the time the shutter is released,and there is a problem in that unsatisfactory photographs may be oftentaken.

On the other hand, in recent years, image processing technologies forperforming digital computation processing on the basis of an imagesignal have progressed rapidly, and as an example thereof, a technologyfor detecting the face of a human being from an image is known. There isa known face detection technology in which, for example, a difference inluminance between two pixels in a face image is learnt as an amount of afeature, an estimated value indicating whether or not a predeterminedregion in an input image is to be computed on the basis of the featureamount, and whether or not the image in the region is a face is finallydetermined on the basis of the estimated value of one or more estimatedvalues (refer to, for example, Japanese Unexamined Patent ApplicationPublication No. 2005-157679 (Paragraph Nos. [0040] to [0052], FIG. 1).

The development of such face detection technologies has progressed to alevel at which such a technology can be installed into a digitalimage-capturing apparatus for performing image capturing using asolid-state image-capturing device, such as a digital still camera.Recently, furthermore, a technology for determining the expression of adetected face has attracted attention. It has been considered that, forexample, the expression of a face of an image-captured person could beevaluated for each captured image from the image signal in which aplurality of frames are continuously captured, so that an appropriateimage can be selected on the basis of the information on thoseevaluations (refer to, for example, Japanese Unexamined PatentApplication Publication No. 2004-46591 (Paragraph Nos. [0063] to [0071],FIG. 3).

BRIEF SUMMARY OF THE INVENTION

In recent years, since competition among makers of digitalimage-capturing apparatuses has become intensified, there has been astrong demand for advancement of such image-capturing apparatuses inorder to increase the product value thereof. As in the problem of theabove-described self-timer function, a captured image is not necessarilysatisfactory to an image-capturing person and an image-captured person.Therefore, it may be said that a function of assisting animage-capturing operation in order to increase such a degree ofsatisfaction is very important for increasing the product value. It hasbeen desired that, in particular, such a function be implemented usingan advanced image processing technology. However, a function ofassisting an image-capturing operation in real time while such animage-capturing operation is being performed has yet to be implemented.

The present invention has been made in view of such problems. It may bedesirable to provide an image-capturing apparatus and method capable ofcapturing an image with high satisfaction for an image-captured personand an image-capturing person.

It may also be desirable to provide an expression evaluation apparatuscapable of capturing an image with high satisfaction for animage-captured person or an image-capturing person, and a program forperforming the processing thereof.

According to an embodiment of the present invention, there is providedan image-capturing apparatus for capturing an image by using asolid-state image-capturing device, the image-capturing apparatus mayinclude a face detector configured to detect a face of a human being onthe basis of an image signal in a period until an image signal obtainedby image capturing is recorded on a recording medium; an expressionevaluation section configured to evaluate the expression of the detectedface and to compute an expression evaluation value indicating the degreeto which the detected face is close to a specific expression in relationto expressions other than the specific expression; and a notificationsection configured to notify notification information corresponding tothe computed expression evaluation value to an image-captured person.

In such an image-capturing apparatus, in a period of time until an imagesignal obtained by image capturing is recorded on a recording medium,the face detector may detect the face of a person from the image signal.The expression evaluation section may evaluate the expression of theface detected by the face detector, and compute an expression evaluationvalue indicating how close the face expression is to a specificexpression in relation to expressions other than the specificexpression. The notification section may notify the image-capturedperson of notification information corresponding to the computedexpression evaluation value.

In the image-capturing apparatus of the present invention, in a periodof time until a captured image signal obtained in image capturing isrecorded on a recording medium, a face of a person may be detected fromthe captured image, the expression of the face may be evaluated, and anexpression evaluation value indicating the degree to which the specificexpression is close to a specific expression in relation to expressionsother than the specific expression may be computed. Then, notificationinformation corresponding to the expression evaluation value may benotified or supplied to the image-captured person. Therefore, it ispossible to allow the image-captured person to recognize whether theexpression of himself/herself is appropriate for image capturing, and asa result, it is possible to prompt the image-captured person to form abetter expression. Therefore, it becomes possible to reliably record animage with high satisfaction for an image-captured person and animage-capturing person on a recording medium.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the configuration of a main part of animage-capturing apparatus according to a first embodiment of the presentinvention;

FIG. 2 is a block diagram showing functions provided in theimage-capturing apparatus so as to implement an expression evaluationmode and an expression response recording mode;

FIG. 3 shows the outline of operations in the expression evaluationmode;

FIG. 4 illustrates the movement of a bar graph indicating a smile score;

FIG. 5 shows an example of a display screen for a smile score using abar graph;

FIG. 6 shows a first example of a screen display for informationcorresponding to an expression evaluation value when a plurality offaces are detected;

FIG. 7 shows a second example of a screen display for informationcorresponding to an expression evaluation value when a plurality offaces are detected;

FIG. 8 conceptually shows information to be generated in advance for thepurpose of expression evaluation, and the flow of informationgeneration;

FIGS. 9A, 9B, and 9C illustrate processes for masking a sample image tobe input at the time of a PCA process;

FIG. 10 conceptually shows a relation between a determination axis andan input image of a face in the pixel space and in the PCA space;

FIG. 11 shows an example of computation when an expression evaluationvalue is output as a numeric value;

FIG. 12 is a flowchart showing the flow of the processing of theimage-capturing apparatus in the expression response recording mode;

FIG. 13 shows the exterior of an image-capturing apparatus according toa second embodiment of the present invention;

FIG. 14 shows the exterior of an image-capturing apparatus according toa third embodiment of the present invention;

FIG. 15 shows the exterior of an image-capturing apparatus according toa fourth embodiment of the present invention; and

FIG. 16 shows the exterior of a PC (personal computer) according to afifth embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention will now be described below withreference to the drawings.

First Embodiment

FIG. 1 is a block diagram showing the configuration of the main part ofan image-capturing apparatus according to a first embodiment of thepresent invention.

The image-capturing apparatus shown in FIG. 1 is implemented as adigital still camera or a digital video camera. The image-capturingapparatus includes an optical block 11, a driver 11 a, animage-capturing device 12, a timing generator (TG) 12 a, an analog frontend (AFE) circuit 13, a camera signal processing circuit 14, a graphicprocessing circuit 15, a display device 16, an image encoder 17, arecording apparatus 18, a microcomputer 19, an input section 20, an LED(Light Emitting Diode) light-emitting section 21, and a sound outputsection 22.

The optical block 11 includes a lens for collecting light from a subjectinto the image-capturing device 12, a driving mechanism for moving thelens in order to perform focusing and zooming, a shutter mechanism, aniris mechanism, and the like. On the basis of a control signal from themicrocomputer 19, the driver 11 a controls driving of each mechanisminside the optical block 11.

The image-capturing device 12 is, for example, a CCD (Charge CoupledDevice) type or CMOS (Complementary Metal Oxide Semiconductor) typesolid-state image-capturing device, is driven on the basis of a timingsignal output from the TG 12 a, and converts incident light from thesubject into an electrical signal. Under the control of themicrocomputer 19, the TG 12 a outputs a timing signal.

The AFE circuit 13 samples and holds the image signal output from theimage-capturing device 12 so as to satisfactorily maintain the S/N(Signal/Noise) ratio by CDS (Correlated Double Sampling) processing,controls gain by AGC (Auto Gain Control) processing, performs A/Dconversion, and outputs digital image data.

The camera signal processing circuit 14 performs, on the image data fromthe AFE circuit 13, AF (Auto Focus), AE (Auto Exposure), a detectionprocess for various kinds of image quality correction processing, and animage quality correction process corresponding to a signal output fromthe microcomputer 19 on the basis of detection information. As will bedescribed later, in this embodiment, the camera signal processingcircuit 14 has a face detection function and a function of extractingdata of a face region.

The graphic processing circuit 15 converts image data output from thecamera signal processing circuit 14 into a signal to be displayed on thedisplay device 16, and supplies the signal to the display device 16.Furthermore, the graphic processing circuit 15 combines information,such as an expression score (to be described later), in an image inresponse to a request from the microcomputer 19. The display device 16is formed of, for example, an LCD (Liquid Crystal Display), and displaysan image on the basis of the image signal from the graphic processingcircuit 15.

The image encoder 17 compresses and codes the image data output from thecamera signal processing circuit 14 and outputs the coded data to therecording apparatus 18. More specifically, the image encoder 17compresses and codes image data for one frame, which is processed by thecamera signal processing circuit 14, in accordance with a coding methodsuch as JPEG (Joint Photographic Experts Group), and outputs the codeddata of a still image. Not only a still image but also data of a movingimage may also be compressed and coded.

The recording apparatus 18 is an apparatus for recording coded data fromthe image encoder 17 as an image file, and is implemented, for example,as an apparatus for driving a portable recording medium such as amagnetic tape or an optical disc, or an HDD (Hard Disk Drive).

The microcomputer 19 includes a CPU (Central Processing Unit), andmemory such as a ROM (Read Only Memory) and a RAM (Random AccessMemory), and centrally controls the image-capturing apparatus byexecuting a program stored in the memory.

The input section 20 outputs, to the microcomputer 19, a control signalcorresponding to operation input to various kinds of input switches by auser. As the input switches, for example, a shutter release button,cross keys used to select various kinds of menus and to set an operationmode, and others are provided.

On the basis of a control signal from the microcomputer 19, the LEDlight-emitting section 21 allows LEDs provided on the exterior surfaceof the image-capturing apparatus to be turned on. Examples of the LEDinclude those indicating that a self-timer function is being operated.

On the basis of a control signal from the microcomputer 19, the soundoutput section 22 outputs sound, such as operation confirmation sound.When an audio data encoder/decoder is provided, reproduction sound whenthe audio data is reproduced may be output.

In this image-capturing apparatus, signals that are photoreceived andphotoelectrically converted by the image-capturing device 12 aresequentially supplied to the AFE circuit 13, whereby a CDS process andan AGC process are performed, and thereafter the signals are convertedinto digital image data. The camera signal processing circuit 14performs an image quality correction process on the image data suppliedfrom the AFE circuit 13, and supplies the image data after processing tothe graphic processing circuit 15, whereby the image data is convertedinto an image signal for display. As a result, an image (camera throughimage) currently being captured is displayed on the display device 16,so that the image-capturing person can confirm the angle of view byviewing the image.

In this state, when an instruction for recording an image is made to themicrocomputer 19 as a result of the shutter release button of the inputsection 20 being depressed, the image data from the camera signalprocessing circuit 14 is supplied to the image encoder 17, whereby acompression and coding process is performed, and the image data isrecorded by the recording apparatus 18. When a still image is to berecorded, image data for one frame is supplied from the camera signalprocessing circuit 14 to the image encoder 17. When a moving image is tobe recorded, processed image data is continuously supplied to the imageencoder 17.

Next, a description will be given of an image-capturing operation modeprovided in the image-capturing apparatus. The image-capturing apparatushas a mode in which, when a still image is to be captured, the face ofan image-captured person is detected from the captured image, theexpression of the face is evaluated, and information indicating thedegree of the evaluation is notified to the image-captured person, and amode in which a shutter is released automatically in response to thedegree of the evaluation and still image data is recorded in therecording apparatus 18. Hereinafter, the former mode will be referred toas an “expression evaluation mode”, and the latter mode will be referredto as an “expression response recording mode”.

The expression evaluation mode serves the role of evaluating theexpression of a face when the face is detected from the captured image,notifying the image-captured person of information corresponding to theevaluation, and prompting the image-captured person to form anexpression more appropriate for image capturing. For example, the degreeof whether or not the expression is a smile is evaluated. Furthermore,in the expression response recording mode, when the evaluation valueexceeds a predetermined value, it is determined that the face of theimage-captured person has become an expression appropriate for imagecapturing, and still image data is automatically recorded. This assiststhat an image with a high degree of satisfaction for the image-capturedperson can be recorded. This embodiment has been described as having twomodes, that is, an expression evaluation mode and an expression responserecording mode. Alternatively, this embodiment may have only theexpression response recording mode.

FIG. 2 is a block diagram showing functions provided in theimage-capturing apparatus so as to implement an expression evaluationmode and an expression response recording mode.

As shown in FIG. 2, the image-capturing apparatus includes, as functionsfor implementing each of the image-capturing operation modes, a facedetector 31, a face image generator 32, an expression evaluation section41, a notification controller 42, and a recording operation controller43. In this embodiment, the face detector 31 and the face imagegenerator 32 are implemented by hardware inside the camera signalprocessing circuit 14. The expression evaluation section 41, thenotification controller 42, and the recording operation controller 43are implemented as functions of software executed by the microcomputer19. However, each of these functions may be implemented by eitherhardware or software. The microcomputer 19 has prestored thereindetermination axis information 44 used for expression evaluationcomputation by the expression evaluation section 41 on a memory such asa ROM provided therein. As will be described later, the determinationaxis information 44 contains coefficient information of a vectorindicating the determination axis of an expression obtained byperforming linear discriminant analysis on the basis of signalcomponents obtained by principal component analysis from sample data ofmany pieces of faces on two expressions.

At this point, the operation of each function shown in FIG. 2 will bedescribed with reference to FIGS. 3 and 4 described below. First, FIG. 3shows the outline of operations in the expression evaluation mode.

In the expression evaluation mode, at first, on the basis of image datathat is obtained by image capturing using the image-capturing device 12and that is transmitted through the camera signal processing circuit 14,the face detector 31 detects the face of an image-captured person fromthe image (step S1). Then, detection information indicating the regionof the detected face is output to the face image generator 32. As inthis embodiment, when a notification is made by displaying informationcorresponding to the expression evaluation value on the display device16, the detection information of the face from the face detector 31 isalso supplied to the notification controller 42 of the microcomputer 19.

As a technique for detecting a face, a well-known technique can be used.For example, a technique disclosed in Japanese Unexamined PatentApplication Publication No. 2005-157679 can be used. In this technique,first, a difference of luminance between two pixels in a face image islearnt, and it is stored in advance as an amount of a feature. Then, asshown in step S1 of FIG. 3, a window W1 of a fixed size is sequentiallyassigned to an input image, whether or not a face is contained in theimage within the window W1 is estimated on the basis of the featureamount, and the estimated value is output. At this time, by sequentiallyreducing the input image and by performing the same process, it ispossible to make an estimation by using the window W1 of a fixed size.Then, the region where a face exists is finally determined on the basisof the estimated value obtained by those operations.

Next, the face image generator 32 extracts data of a region Af of thedetected face (step S2). Then, the face image generator 32 converts theextracted image data into image data of a fixed size, normalizes it, andsupplies the image data to the expression evaluation section 41 (stepS3).

At this point, in this embodiment, as examples of detection informationof the face, which is output from the face detector 31, the position(for example, the coordinate of the left end, hereinafter referred to as“position information of the face”) of a detection frame of a rectanglesurrounding the periphery of the face, and the size of the detectionframe (for example, the number of pixels in each of the horizontal andvertical directions, hereinafter referred to as “size information of theface”) are assumed to be output. In this case, the face image generator32 accesses the memory (RAM) in which image data for which the face isto be detected is temporarily stored, and reads only the data of theregion corresponding to the position information and the sizeinformation of the face from the face detector 31.

The extracted image data is normalized by being subjected to resolutionconversion as image data of a fixed size (resolution). The image sizeafter the normalization becomes a size that becomes a processing unitwhen the expression evaluation section 41 evaluates the expression ofthe face. In this embodiment, as an example, the size is set at 48×48pixels.

As the image extraction function and the resolution conversion functionprovided in the face image generator 32, the same functions that aretypically provided for the camera signal processing circuit 14 of therelated art for the purpose of detection and generation of an outputimage can also be used.

Next, on the basis of the normalized image data of the face from theface image generator 32 and the determination axis information 44 thatis stored in advance, the expression evaluation section 41 performs anoperation for evaluating the degree of appropriateness of the expressionof the face and outputs the expression evaluation value (step S4). Theexpression evaluation value indicates the degree to which the expressionof the face is close to one of the two expressions. For example, as twoexpressions, “smile” and “usual expression” are used. The higher theexpression evaluation value, the higher the degree to which theexpression is estimated to be a “smile” rather than “usual expression”.The method of computing the expression evaluation value will bedescribed later.

Next, the notification controller 42 notifies the image-captured personof information corresponding to the expression evaluation value outputfrom the expression evaluation section 41 (step S5). For example, theinformation corresponding to the expression evaluation value isdisplayed via the graphic processing circuit 15 on the display device 16oriented toward the image-captured person side. In this case, displaymay be performed such that a face to be evaluated is specified withinthe display device 16 on the basis of the position and the sizeinformation on the face supplied from the face detector 31. A differencein the expression evaluation value may be notified on the basis of achange in the luminance, a change in the blinking speed, and a change inthe color of the LED light-emitting section 21 by using the LEDlight-emitting section 21. Alternatively, a notification may be made byoutputting sound that differs according to the expression evaluationvalue via the sound output section 22.

In the following description, the expression evaluation section 41 isassumed to evaluate, as an example, the degree about whether theexpression of the face is a smile or expressionless. In this embodiment,in particular, the information corresponding to the expressionevaluation value is notified to the image-captured person by displayingit on the display device 16, the display screen thereof being orientedtoward the image-captured person side. In FIG. 3, an example is shown inwhich a bar graph indicating a “smile score”, which is a valuecorresponding to the expression evaluation value, is displayed on thedisplay device 16.

FIG. 4 illustrates the movement of a bar graph indicating a smile score.

As shown in FIG. 4, the stronger the degree that the expression of theface is a smile, the higher the expression evaluation value becomes, andthe higher the degree that the expression of the face is a usualexpression, the lower the expression evaluation value becomes. The smilescore shown in the bar graph varies continuously or in a step-likemanner in proportion to the expression evaluation value. The bar graphis displayed on the display device 16 oriented toward the image-capturedperson side. The image-captured person can recognize whether or nothis/her expression is a smile appropriate for image capturing by viewingthe bar graph in real time at the time of image capture. As a result,the bar graph functions to assist the image-capturing operation so thatthe image-captured person is prompted to form an expression appropriatefor image capturing and a more satisfactory image can be captured. Aswill be described later, specific character information for prompting animage-captured person with a low expression evaluation value to form asmile may be displayed.

At this point, when the mode is set to an “expression response recordingmode”, the expression evaluation section 41 performs control so that,when the expression evaluation value exceeds a predetermined thresholdvalue, the shutter is released automatically, that is, the capturedimage is recorded. In FIG. 2, the recording operation controller 43 is ablock for controlling the recording operation of captured image data. Ina normal image-capturing operation, when the recording operationcontroller 43 detects that the shutter release button of the inputsection 20 is depressed, the recording operation controller 43 performscontrol so that each section of the image-capturing apparatus performsan appropriate image-capturing operation (for example, exposureoperation and signal processing operation) at the time of recording,thereby the image encoder 17 encodes the captured image data, and thecoded data is recorded by the recording apparatus 18. Then, when theexpression evaluation value exceeds the predetermined threshold value,the expression evaluation section 41 requests the recording operationcontroller 43 to perform an operation of recording image data.

As a result, when a face is detected from the captured image and theexpression of the face is evaluated to be appropriate for imagecapturing (here, when the degree to which the face expression is closeto a smile becomes high), the captured image at that time isautomatically recorded. Therefore, when compared with the self-timerfunction of the related art (that is, the function of recording acaptured image after a fixed period of time has elapsed from when theshutter release button is depressed), it becomes possible to reliablycapture an image in which the image-captured person has a satisfactoryexpression, and it is possible to increase the degree of satisfaction ofthe image-captured person and the image-capturing person.

Next, a description will be given below of an example of a specificdisplay screen for a smile score on the display device 16.

FIG. 5 shows an example of the screen display for a smile score using abar graph.

In FIG. 5, a digital video camera 100 is assumed as an image-capturingapparatus. In the digital video camera 100, a display device 16 forconfirming the angle of view is provided on the side surface of a cameramain unit 101. In the digital video camera 100 of such a configuration,generally, the angle of the display screen of the display device 16 isvariable, so that, as shown in FIG. 5, the display screen can beoriented in the direction in which an image-capturing lens 102 isprovided, that is, in the direction of the image-captured person. In theexpression evaluation mode and the expression response recording mode,the digital video camera 100 is used in a state in which the displayscreen of the display device 16 is oriented toward the image-capturingperson side, and information corresponding to the expression evaluationvalue is displayed together with the captured image of the subject.

In the example of the screen display of FIG. 5, a smile score displaysection 202 is combined on a captured image containing a face 201, andis displayed. On the smile score display section 202, a smile scorecorresponding to the expression evaluation value is displayed as a bargraph 203 and also, the smile score is displayed as a numeric value on anumeric value display section 204. In the expression response recordingmode, a boundary display icon 205 indicating the boundary of the smilescore when a captured image is automatically recorded is displayed. Inthis example, a threshold value in the form of a numeric value isdisplayed on the boundary display icon 205.

In the example of FIG. 5, together with the smile score display section202, a face display frame 206 is displayed around the face 201corresponding to the smile score, so that the face 201 for which a smileis to be evaluated is displayed in an easily understandable manner.Furthermore, a character display section 207 for displaying a characterthat differs in accordance with the expression evaluation value isprovided in the vicinity of the face display frame 206, so that theimage-captured person is prompted to form, the lower the degree of beinga smile, a stronger smile by using a character.

In the expression response recording mode, the threshold value of theexpression evaluation value when a captured image is recorded may be setby a user as desired, so that the image-captured person can freelydetermine to what degree he/she wishes a face expression is close to asmile before a captured image is recorded. In the example of FIG. 5, forexample, by the user depressing the direction key in a side-to-sidedirection (not shown) provided in the input section 20, the thresholdvalue of the expression evaluation value is changed, the boundarydisplay icon 205 moves in the side-to-side direction, and it is possiblefor the user to visually recognize a smile score corresponding to thethreshold value of the expression evaluation value. In this case, thedirection key operated in the side-to-side direction may automaticallyserve as a key for setting a threshold value of an expression evaluationvalue when the mode is set to the expression response recording mode,thereby making it possible to increase the ease of operation for theuser.

The changing of the threshold value of the expression evaluation valueis not limited to the above-described method, and may be performed froma dedicated setting screen selected from the menu screen. Alternatively,a dedicated operation key may be provided to change the threshold valueof the expression evaluation value. When the display device 16 is of atouch panel type, for example, the threshold value may be changed byallowing a finger to contact a key image displayed on the display device16. Furthermore, a threshold value may be changed by moving a finger inthe side-to-side direction with the finger in contact with the boundarydisplay icon 205 of FIG. 5.

When a plurality of faces are detected from within the image-capturescreen, an expression evaluation value may be computed for each of thosefaces, and information corresponding to those values may be displayed onthe display device 16. FIG. 6 shows a first example of a screen displayfor information corresponding to an expression evaluation value when aplurality of faces are detected.

FIG. 6 shows an example in which two faces 211 and 212 are detected. Inthe faces 211 and 212, face display frames 213 and 214 are displayedaround the face regions, respectively, and character display sections215 and 216 are provided in the vicinity thereof, respectively. The linetype of each of the face display frames 213 and 214 is changed inaccordance with the expression evaluation value for each of the faces211 and 212, and a different character corresponding to the expressionevaluation value is displayed on the character display sections 215 and216.

In the example of FIG. 6, the face 211 is evaluated to have asufficiently strong smile, but the face 212 is evaluated to have aninsufficient smile. For example, the expression evaluation value of theface 211 has reached a threshold value for automatic recording, but theexpression evaluation value of the face 212 indicates a value a littlelower than the threshold value. At this time, such differences in theevaluation state are notified to the corresponding image-captured personwith a face display frame 213 for the face 211 being indicated by asolid line and a face display frame 214 for the face 212 being indicatedby a broken line and also, character information for prompting that asmile be formed stronger is displayed on the character display section216. In this example, differences in the expression evaluation value areshown using the line type of the face display frames 213 and 214. Inaddition, differences in the expression evaluation value may be notifiedby, for example, differences in the luminance, the color, the thicknessof the face display frame.

FIG. 7 shows a second example of a screen display for displayinginformation corresponding to an expression evaluation value when aplurality of faces are detected.

Also, in the example of FIG. 7, similarly to FIG. 6, two faces, that is,faces 211 and 212, are detected. The face 211 is evaluated to have asufficiently strong smile, and the face 212 is evaluated to have aninsufficiently strong smile. Furthermore, in the example of FIG. 7,differences in the expression evaluation value are notified to theimage-captured person by showing symbols 217 and 218 corresponding tothe expression evaluation value in the vicinity of each of the regionsof the faces 211 and 212, respectively.

In the manner described above, as a result of performing notification ofinformation corresponding to an expression evaluation value by using adisplay device, it is possible to notify the image-captured person ofinformation corresponding to an expression evaluation value by variousmethods, such as by displaying a smile score corresponding to anexpression evaluation value using a bar graph or a numeric value, theline type, color, brightness, and the like of a face display frame beingchanged in accordance with an expression evaluation value, or acharacter for prompting the image-captured person to smile in accordancewith an expression evaluation value being displayed in the vicinity of aface. In particular, in the case of a digital video camera, since anotification can be made using a display device in which the displayscreen orientation is variable, which is heretofore provided, it ispossible to reliably record an image with a high satisfaction for theuser without causing a large increase in the development/manufacturingcost due to the changing of the basic configuration of the camera.

In the foregoing, a digital video camera is used as an example in whicha display device in which the display screen orientation is variable isinstalled. Some digital still cameras provided with a display device forconfirming the angle of view on a side opposite to an image-capturinglens are such that the display screen orientation is variable. If thedisplay screen thereof can be oriented toward the image-captured personside, the above-described display image can be displayed, making itpossible to notify the image-captured person of informationcorresponding to the expression evaluation value.

Next, a description will be given of an expression evaluation methodused in this image-capturing apparatus.

FIG. 8 conceptually shows information to be generated in advance for thepurpose of expression evaluation, and the flow of informationgeneration.

In this embodiment, as a method for evaluating an expression, aso-called “Fisher linear discriminant analysis” is used. In this method,first, many sample images of faces each having two expressions areprovided in advance. On the basis of the data of these sample images, byconsidering as a two-class problem between two expressions, adetermination axis A_(d) by which the two expressions are satisfactorilydetermined by linear discriminant analysis (LDA) is formed in advance.Then, when an expression evaluation is to be made, the inner product ofthe input data of the face image and the determination axis A_(d) isdetermined to compute the expression evaluation value.

As shown in FIG. 8, in this embodiment, a sample image Ps of a smile anda sample image Pn of a usual expression are used. The sample images Psand Pn are provided in advance as images that are normalized to a fixedsize of, for example, 48×48 pixels. Then, the data of these sampleimages is handled as vector data of 48 . . . × . . . 48 dimensions, andan LDA process is performed. However, the vector space becomes a spacewith a very large dimension having 48 . . . × . . . 48 coordinate axes.Therefore, prior to the LDA process, principal component analysis (PCA)is performed on those vector data so as to be converted(dimension-compressed) into data with a low dimension space thatefficiently indicates only the features of the face.

In this PCA process, first, it is considered to obtain M axes so thatthe variance among sample image groups of N dimensions (N=48 . . . × . .. 48) of input M (for example, M=300) becomes a maximum. Such axes aredetermined as solutions (intrinsic vector) of an intrinsic value problemof a covariance matrix of an image group, and by extracting only vectorcomponents having a comparatively large coefficient as principalcomponents, the data can be compressed into data of N′ dimensions(N>>N′) of only vector components suitable for showing the features ofthe face. It is known that, for example, by setting N′=approximately 40,sufficient accuracy can be maintained for the determination of the faceexpression. By excluding several components starting from the largecoefficient from among the principal components obtained by the PCAprocess, the number of dimensions can be reduced further and the burdenof the next PCA process can be reduced while maintaining the expressiondetermination accuracy.

FIGS. 9A, 9B, and 9C illustrate processing for masking a sample image tobe input at the time of a PCA process.

As shown in FIG. 9A, when a PCA process is performed by using as it is asample image P in which a face 221 is image-captured in a rectangularregion of a predetermined size, there can be a case in which appropriateprincipal components cannot necessarily be selected due to thebackground of the face 221, the hair, and the like. For this reason, asshown in FIG. 9B, by applying a mask 222 to the region other than theface 221, the sample image is converted into a sample image P1 in whichonly the face region remains as much as possible, and a PCA process isperformed in a state in which the information density of the face isincreased, making it possible to perform dimension compression withhigher accuracy. Furthermore, the mouth within the face region isgreatly changed depending on the expression and often becomes adisturbance element. Therefore, as shown in FIG. 9C, by performing a PCAprocess using a sample image P2 in which a mask 223 is applied even tothe region of the mouth, it is possible to further increase the accuracyof dimension compression.

Referring back to FIG. 8, a description will be given below. Each sampleimage of a smile that is dimension-compressed by a PCA process and ausual expression is converted into vector data in the partial space (PCAspace S_(PCA)) having only coordinate axes that efficiently show thefeatures of a face. FIG. 8 schematically shows a state in which thesample image Ps of a smile and a sample image Pn of a usual expressionare projected on the PCA space S_(PCA). As shown in FIG. 8, it may besaid that sample images having the same expression exist comparativelyclose to each other in the PCA space S_(PCA). Therefore, a sample imagegroup of these expressions is regarded as two clusters CLs and CLn, anda projection axis (determination axis A_(d)) that best separates theclusters is formed by an LDA process. Such a determination axis A_(d) iscalled a “Fisher's projection axis”.

In the LDA process, in general, a determination axis is determined suchthat inter-class and intra-class variances projected on the intrinsicvector of N′ dimensions become a maximum. That is, an intrinsic vectorcorresponding to the maximum intrinsic value of each of inter-class andintra-class covariance matrixes is determined, and the intrinsic vectoris set as a vector (Fisher vector) in the determination axis A_(d). Therelation between a covariance matrix and an intrinsic value and therelation between a covariance matrix and an intrinsic vector are shownin equations (1) and (2), respectively.R_(B){right arrow over (μ)}=λR_(W){right arrow over (μ)}  (1)

R_(W) ⁻¹R_(B){right arrow over (μ)}=λ{right arrow over (μ)}  (2)

-   -   (R_(W): Intra-class covariance matrix, R_(B): Inter-class        covariance matrix, λ: Intrinsic value)    -   Fisher vector→Intrinsic vector corresponding to maximum        intrinsic value

For the computation of the reverse matrix, the intrinsic value, and theintrinsic vector of the left side of equation (2), an LU (Lower-Upper)decomposition method, a QR decomposition method (Q: orthogonal matrix,R: upper triangular matrix), and a Gaussian elimination method can beused, respectively. The expression evaluation section 41 prestoresinformation, such as a coefficient of each component of a Fisher vector,as information (determination axis information 44) on the determinationaxis A_(d) obtained in the above-described manner in a ROM or the like.

FIG. 10 conceptually shows relations between a determination axis and aninput image of a face in the pixel space and in the PCA space.

The basic procedure of expression determination using theabove-described determination axis A_(d) is as follows. At first, imagedata of a face detected from a captured image is subjected to a PCAprocess, and principal components are extracted. Then, the expression ofthe face image, as shown in the PCA space S_(PCA) of FIG. 10, isevaluated as projection components of the PCA-processed face imagevector (input face image vector) with respect to the determination axisA_(d). That is, the computation of the expression evaluation valueE_(exp) can be computed by the inner product of the input face imagevector and the Fisher vector (see equation (5-1)).

$\begin{matrix}{{\overset{\rightarrow}{P}}_{{in},{{px}\; 1}} \approx {{1.45{\overset{\rightarrow}{\mu}}_{1}} + {0.86{\overset{\rightarrow}{\mu}}_{2}} + {0.64{\overset{\rightarrow}{\mu}}_{3}} + \ldots + {0.05{\overset{\rightarrow}{\mu}}_{N^{\prime}}} + C}} & (3) \\{{\overset{\rightarrow}{A}}_{d,{{px}\; 1}} \approx {{0.98{\overset{\rightarrow}{\mu}}_{1}} + {0.45{\overset{\rightarrow}{\mu}}_{2}} + {0.38{\overset{\rightarrow}{\mu}}_{3}} + \ldots + {0.09{\overset{\rightarrow}{\mu}}_{N^{\prime}}} + C}} & (4) \\\begin{matrix}{E_{\exp} = {{{\overset{\rightarrow}{P}}_{{in},{pea}} \cdot {\overset{\rightarrow}{A}}_{d}} = {\begin{bmatrix}1.45 \\0.86 \\0.64 \\\vdots \\0.05\end{bmatrix} \cdot \begin{bmatrix}0.98 \\0.45 \\0.38 \\\vdots \\0.09\end{bmatrix}}}} \\{= {\left\lbrack {{\overset{\rightarrow}{P}}_{{in},{{px}\; 1}} - C} \right\rbrack \cdot \left\lbrack {{\overset{\rightarrow}{A}}_{d,{{px}\; 1}} - C} \right\rbrack}} \\{\begin{pmatrix}{{\overset{\rightarrow}{P}}_{{in},{{px}\; 1}}\text{:}\mspace{14mu}\begin{matrix}{{Input}\mspace{14mu}{face}\mspace{14mu}{image}\mspace{14mu}{vector}\mspace{14mu}{in}} \\{{pixel}\mspace{14mu}{space}}\end{matrix}} \\{{\overset{\rightarrow}{A}}_{d,{{px}\; 1}}\text{:}\mspace{14mu}{Fisher}\mspace{14mu}{vector}\mspace{14mu}{in}\mspace{14mu}{pixel}\mspace{14mu}{space}} \\{{\overset{\rightarrow}{\mu}}_{1},\ldots\mspace{14mu},{{\overset{\rightarrow}{\mu}}_{N^{\prime}}\text{:}\mspace{14mu}{Vector}\mspace{14mu}{of}\mspace{14mu}{principal}\mspace{14mu}{components}}} \\{E_{\exp}\text{:}\mspace{14mu}{Expression}\mspace{14mu}{evaluation}\mspace{14mu}{value}} \\{{\overset{\rightarrow}{P}}_{{in},{pea}}\text{:}\mspace{14mu}\begin{matrix}{{Input}\mspace{14mu}{face}\mspace{14mu}{image}\mspace{14mu}{vector}} \\{{after}\mspace{14mu}{PCA}\mspace{14mu}{process}}\end{matrix}}\end{pmatrix}}\end{matrix} & \begin{matrix}\left( {5\text{-}1} \right) \\\left( {5\text{-}2} \right)\end{matrix}\end{matrix}$

The information on the Fisher vector can also be converted intoinformation in the pixel space S_(pxl) (the space of the dimensionpossessed by the original image data before the PCA process). Equation(3) shows an input face image vector as a vector in the pixel spaceS_(pxl), and equation (4) shows a Fisher vector as a vector in the pixelspace S_(pxl). FIG. 10 conceptually shows these relations. As shown inequation (3), equation (4), and FIG. 10, vector components other thanprincipal components μ₁ to μ_(N′) obtained by the PCA process can beapproximated using a constant C as an average value of all the inputimages. Therefore, the inner product computation shown in equation (5-1)can be equivalently expressed as an inner product computation of vectorsin the pixel space S_(pxl), as shown in equation (5-2).

In equation (5-2), since the result of the subtraction between theFisher vector components and the constant C in the pixel space S_(pxl)can be computed in advance, the expression evaluation section 41 hasstored therein the subtraction result and the constant C in advance asthe determination axis information 44. Then, when the vector of the faceimage detected from the captured image is given, the inner productcomputation of equation (5-2) is performed without performing a PCAprocess on the vector. In the evaluation value computation for one faceusing equation (5-2), a maximum number of times subtractions,multiplications, and additions are performed is 48 . . . × . . . 48only. Moreover, in practice, only the computation of a coefficientcorresponding to the 40 or so principal components μ₁ to μ_(N′) isperformed. Therefore, when compared with the case in which the innerproduct computation of vectors in the PCA space S_(PCA) is performed,the number of computations can be greatly reduced without decreasing theaccuracy of expression evaluation, and the expression evaluation valueE_(exp) can be easily computed in real time in a state of angle of viewmatching before the captured image is recorded.

In such a computation method, for example, even when compared with thecase in which an expression is evaluated by matching between a templateof many face images and the detected face image, it is possible toperform expression evaluation with a low processing load and with highaccuracy. When matching using a template is to be performed, usually, itis necessary to further extract parts, such as the eye or the mouth,from the detected face image and to perform a matching process for eachpart. In comparison, in the method of this embodiment, after the data ofthe detected face image is normalized to a fixed size, the face image isreplaced with vector information and can be applied to an inner productcomputation as it is (or only partially applying a mask), and the innerproduct computation becomes a simple computation composed ofsubtractions, multiplications, and additions of approximately 40dimensions as described above.

FIG. 11 shows an example of computation when an expression evaluationvalue is output as a numeric value.

In this embodiment, as an example, on the basis of the result of the PCAprocess for a sample image, an average of each distribution of faceimages of smiles and usual expressions in the PCA space is determined,and a projection point of these averages with respect to thedetermination axis A_(d) is determined in advance. Then, by using themiddle point of the projection points of each average as a reference,the expression evaluation value E_(exp) is converted into a numericvalue. That is, as shown in FIG. 11, the distance between the projectionpoint of the input face image with respect to the determination axis andthe middle point of the average projection point is set as an expressionevaluation value E_(exp), and the side where sample images of smiles aredistributed is set to a positive numeric value. As a result, as to whichof the smile and the usual expression the detected image of the face isclose to can be output as a continuous numeric value, and the higher theexpression evaluation value E_(exp), the stronger smile the face isevaluated to have.

Next, a description will be given of the processing procedure of animage-capturing apparatus operating in the expression response recordingmode, the processing procedure being summarized in a flowchart. FIG. 12is a flowchart showing the flow of the processing of the image-capturingapparatus operating in the expression response recording mode.

[Step S11] The face detector 31 detects a face from the data of acaptured image and outputs the position information and the sizeinformation of all the detected faces to the face image generator 32 andthe notification controller 42.

[Step S12] On the basis of the position information and the sizeinformation of the face from the face detector 31, the face imagegenerator 32 extracts data in a region of each face detected from thedata of the captured image.

[Step S13] The face image generator 32 normalizes the data of each ofthe extracted face regions into data of a predetermined number of pixels(here, 48×48 pixels), applies masking to a region for which anexpression does not need to be detected, and outputs the image dataafter processing to the expression evaluation section 41.

[Step S14] The expression evaluation section 41 reads the determinationaxis information 44, and computes the inner product of vectors obtainedfrom one face image supplied from the face image generator 32 and thevector components of the determination axis in order to compute anexpression evaluation value. The computed expression evaluation valueis, for example, temporarily stored in a RAM or the like.

[Step S15] The expression evaluation section 41 determines whether ornot the expression evaluation process has been performed on all thedetected faces. When all the detected faces have not been processed,step S14 is performed again on another face, and when all the detectedfaces have been processed, step S16 is performed.

[Step S16] On the basis of the expression evaluation value computed instep S15 and the position information and the size information of theface corresponding to the expression evaluation value, the notificationcontroller 42 outputs the expression information such as the smile scoreand the display frame to the graphic processing circuit 15 and displaysthem in such a manner as to be combined on the display device 16.

[Step S17] The expression evaluation section 41 determines whether ornot the expression evaluation values for all the faces, which arecomputed in step S14, exceed a threshold value. When there is anexpression evaluation value that does not exceed the threshold value,the process returns to step S11, where the expression evaluation section41 instructs the camera signal processing circuit 14 to detect a face,thereby starting a process for detecting the next face and an expressionevaluation process. When all the expression evaluation values exceed athreshold value, step S18 is performed.

[Step S18] The expression evaluation section 41 requests the recordingoperation controller 43 to record the data of the captured image in therecording apparatus 18. as a result, a recording process is performed onthe captured image, and the coded image data after the processing isrecorded in the recording apparatus 18.

As a result of the above processing, expression evaluation values arecomputed for all the detected faces, and information corresponding tothe expression evaluation value is notified as display information tothe image-captured person, thereby making it possible to prompt theimage-captured person to form an expression appropriate for imagecapturing. When all the image-captured persons have formed expressionsappropriate for image capturing, the data of the captured image isautomatically recorded. Therefore, it is possible to reliably record animage with a high degree of satisfaction for the image-captured personand the image-capturing person.

The determination criterion in step S17 is only an example, and controlis not necessarily performed in such a manner that image data isrecorded when all the expression evaluation values exceed a thresholdvalue. For example, when expression evaluation values of a fixed ratioof faces among the detected faces exceed a threshold value, image datamay be recorded. Alternatively, image data may be recorded whenexpression evaluation values of a fixed number of faces exceed athreshold value, so that thereafter, expression evaluation is preventedfrom being performed on an inadvertently image-captured unwanted face.

In the expression response recording mode, when an expression evaluationvalue exceeds a predetermined threshold value, a captured image isautomatically recorded. In addition, for example, when a shutter releasebutton is depressed by an image-capturing person, the expression of theimage-captured person is evaluated after a fixed period of time haselapsed, and a captured image may be automatically recorded when theimage-captured person forms an expression appropriate for imagecapturing. In this case, for example, when the depression of the shutterrelease button is detected, the microcomputer 19 needs only to startcounting time and start processing illustrated in FIG. 12 when a fixedperiod of time has passed. As a result of such processing, theimage-capturing person who depressed the shutter release button canreliably move into an image-capturing range, and the ease of operationcan be improved.

In the foregoing description, two expressions, that is, a “smile” and an“usual expression”, are defined, in addition to the degree to which theexpression is close to a smile. In addition, a determination may beperformed that an expression lies between a “smile” and expressionsother than a “smile” (referred to as a non-smile). Non-smile expressionsmay include a plurality of expressions that are not a smile, such as aserious expression, a weeping expression, and an angry expression. Inthis case, a group of non-smile expressions, hereafter, “non-smiles”, isdetermined on the basis of the average of sample images of facescorresponding to the plurality of expressions, and a determination axisfor an LDA process is computed on the basis of the group of “non-smiles”and a group of “smiles”.

Furthermore, the expression evaluation value does not necessarily needto be a measure of closeness to one particular expression, such as a“smile”. For example, by considering a plurality of specificexpressions, such as a “smile” and a “serious expression”, to beexpressions appropriate for image capturing, the expression evaluationvalue may indicate how close the expression is to any of the pluralityof expressions. In this case, also, a group of “expressions appropriatefor image capturing” may be determined on the basis of the average ofsample images of faces corresponding to the plurality of expressions,and a determination axis for an LDA process may be computed on the basisof the group of “expressions appropriate for image capturing” and thegroup of “expressions inappropriate for image capturing”.

Second Embodiment

FIG. 13 shows the exterior of an image-capturing apparatus according toa second embodiment of the present invention.

In this embodiment, information corresponding to an expressionevaluation value is notified to an image-captured person by using a partof the LED light-emitting section 21 within the configurationillustrated in FIG. 1. In an image-capturing apparatus 110 shown in FIG.13, a dedicated LED light-emitting section 21 a for notifyinginformation corresponding to an expression evaluation value is providedon the side where an image-capturing lens 111, a flash-light-emittingsection 112, and the like are installed. The LED light-emitting section21 a is provided with a plurality of LEDs 21 b to 21 f in one line.Information (here, a smile score) corresponding to the expressionevaluation value is notified to the image-captured person on the basisof the number of LEDs emitting light among them. With such aconfiguration, it is also possible for an image-capturing apparatus suchas a digital still camera not having a display device for which thedisplay screen orientation is variable to notify informationcorresponding to the expression evaluation value to the image-capturedperson and to assist an image-capturing operation so that an appropriateimage can be recorded. Furthermore, by using a small light-emittingdevice, such as an LED, the enlargement of the main unit of theimage-capturing apparatus can be suppressed to a minimum.

Furthermore, in the LED light-emitting section 21 a, the LED (the LED 21f in FIG. 13) on the farthest side may be set to indicate a smile scorewhen a captured image is automatically recorded, and this LED may emitlight of a color and at a luminance differing from those of the otherLEDs. As a result, it is possible to clearly notify a smile score whenautomatic recording is performed to an image-captured person. Also, itis possible for an image-captured person to recognize the fact thatautomatic recording has been performed.

Third Embodiment

FIG. 14 shows the exterior of an image-capturing apparatus according toa third embodiment of the present invention.

In an image-capturing apparatus 120 shown in FIG. 14, an LEDlight-emitting section 21 g having only one LED is provided on a sidewhere an image-capturing lens 121, a flash light-emitting section 122,and the like are installed. In such an LED light-emitting section 21 g,it is possible to notify a smile score to the image-captured person onthe basis of an expression evaluation value, for example, by changingthe blinking speed of an LED or by changing the luminance/color of anLED. Control is possible so that, for example, as the expressionevaluation value increases, the color of the LED is gradually changedfrom red, to green, and then to blue, or the LED is made to emitbrighter light. In the manner described above, by using only one LED,the enlargement of the main unit of the image-capturing apparatus can beprevented more markedly.

When the image-capturing apparatus 120 has a self-timer function of therelated art, an LED used when the self-timer operates can also be usedin the operation of evaluating an expression. For example, when theself-timer operates, the blinking speed of the LED is graduallyincreased as time passes from when the shutter release button isdepressed until recording is performed. Then, in the expressionevaluation mode and in the expression response recording mode, thehigher the expression evaluation value, the more the blinking speed ofthe LED is increased. With such a configuration, it is possible tonotify the image-captured person of information corresponding to theexpression evaluation value without changing the basic configuration andthe exterior of the image-capturing apparatus of the related art. Thedouble-function light-emitting section is not limited to the function ofa self-timer section and can also be used as a light-measuringlight-emitting section during exposure control. However, in this case,it is necessary for the section to be capable of emitting visible lightat least at the time of expression evaluation.

Fourth Embodiment

FIG. 15 shows the exterior of an image-capturing apparatus according toa fourth embodiment of the present invention.

In each of the above-described embodiments, information corresponding toan expression evaluation value is visually notified. In comparison, inthis embodiment, information corresponding to an expression evaluationvalue is notified using sound by using the sound output section 22 shownin FIG. 2. In an image-capturing apparatus 130 shown in FIG. 15, aspeaker 22 a is provided on a side where an image-capturing lens 131 isinstalled, and sound that differs depending on an expression evaluationvalue is reproduced and output. For sound to be output, for example, inthe same manner as for character information shown in FIGS. 5 and 6,sound is used such that the lower the degree of being a smile, the morethe image-captured person is prompted to form a strong smile. In thiscase, the image-capturing apparatus 130 may store data of sound to bereproduced in advance in such a manner as to correspond to an expressionevaluation value in a step-like manner. Furthermore, a method of varyingthe pitch of sound and the output intervals of sound according to theexpression evaluation value, or a method of outputting sound of melody,which differs depending on the expression evaluation value, may beadopted. Notification using sound and notification using visualinformation may be used in combination.

Fifth Embodiment

FIG. 16 shows the exterior of a PC (personal computer) according to afifth embodiment of the present invention.

The expression evaluation function, the function of notifyinginformation on the basis of an expression evaluation value, and thefunction of automatically recording an image on the basis of anexpression evaluation value in each of the above-described embodimentscan also be implemented in various kinds of computers as in a PC 140shown in FIG. 16. In FIG. 16, as an example, a notebook-type PC 140 inwhich a display device 141 including an LCD, a keyboard 142, and themain unit are integrally configured is shown. In the PC 140, forexample, an image-capture unit 143 is integrally provided in the upperend part of a display device 141, so that the side of the user who isoperating the PC 140 can be image-captured. The image-capture unit 143may be connected to the outside, for example, via a communicationinterface such as a USB (Universal Serial Bus).

In such a computer, the above-described functions are implemented in thecomputer by the computer executing a program describing processingcontent of each of the functions. The program describing the processingcontent can be recorded in advance on a computer-readable recordingmedium. Examples of computer-readable recording media include a magneticdisk, an optical disc, a magneto-optical recording medium, and asemiconductor memory.

When this program is to be distributed, for example, portable recordingmedia, such as optical discs on which programs are recorded, are sold.Alternatively, a program can be stored in advance in a storage apparatusof a server computer, and a program can also be transferred from aserver computer to another computer.

The computer that executes a program stores, for example, a programrecorded on a portable recording medium or a program extracted from theserver computer, in the storage apparatus of the computer. Then, thecomputer reads a program from its storage apparatus and performsprocessing in accordance with the program. The computer can alsodirectly read a program from a portable recording medium, and canperform processing in accordance with the program. The computer can alsoperform processing in accordance with received programs one by one eachtime a program is transferred from a server computer.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Although the invention herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent invention. It is therefore to be understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present invention as defined by the appended claims.

1. An image-capturing apparatus for capturing an image using asolid-state image-capturing device, the image-capturing apparatuscomprising: a face detector configured to detect a face of a human beingbased on an image signal in a period until an image signal obtained byimage capturing is recorded on a recording medium; an expressionevaluation section configured to evaluate the expression of the detectedface and to compute an expression evaluation value indicating the degreeto which the detected face is closer to a specific expression than toexpressions other than the specific expression; and a notificationsection configured to output notification information corresponding tothe computed expression evaluation value, wherein the notificationsection notifies, as the notification information, information thatvisually indicates the magnitude of the expression evaluation value toan image-captured person side, and the notification section comprises alight-emitting section used to provide both a function of indicating themagnitude of the expression evaluation value and another function. 2.The image-capturing apparatus according to claim 1, wherein thenotification section comprises a plurality of light-emitting sections,the light-emitting surface of each of the light-emitting sections beingoriented toward the image-captured person side, and indicates themagnitude of the expression evaluation value by changing the number oflight-emitting sections emitting light.
 3. The image-capturing apparatusaccording to claim 1, wherein the expression evaluation section haspre-stored therein information on a determination axis obtained byperforming linear discriminant analysis based on a group of sampleimages of faces each corresponding to one of the specific expression andexpressions other than the specific expression, and computes theexpression evaluation value based on the magnitude of projectioncomponents of a vector based on an image signal of the face detected bythe face detector with respect to a vector on the determination axis,the determination axis is computed based on data such that data of avector based on the sample images is dimension-compressed by principalcomponent analysis, and the expression evaluation section has pre-storedtherein data of a vector in the determination axis as data of a vectorhaving a dimension before being dimension-compressed, and computes theexpression evaluation value by computing an inner product of the vectorhaving a dimension before being dimension-compressed and a vector basedon an image signal of the face detected by the face detector.
 4. Theimage-capturing apparatus according to claim 1, wherein the notificationsection is configured to output notification information correspondingto the computed expression evaluation value to an image-captured person.5. The image-capturing apparatus according to claim 1, wherein thelighting surface of the light-emitting section being oriented toward theimage-captured person side, and displays the magnitude of the expressionevaluation value as a change in the color of the light-emitting section,a change in the luminance thereof, or a change in the blinking speedthereof.
 6. The image-capturing apparatus according to claim 5, furthercomprising: a self-timer recording controller configured to performcontrol so that an image signal obtained by image capturing is coded andautomatically recorded on a recording medium after a fixed period oftime from when the depression of a shutter release button is detected;and a self-timer light-emitting section configured to notify animage-captured person of the fact that the self-timer is being operatedfrom when the shutter release button is depressed until when an imagesignal is recorded on the recording medium during control by theself-timer recording controller, wherein the light-emitting sectionconfigured to indicate the magnitude of the expression evaluation valueis also used for the self-timer light-emitting section.
 7. Theimage-capturing apparatus according to claim 1, wherein the expressionevaluation section has pre-stored therein information on a determinationaxis obtained by performing linear discriminant analysis based on agroup of sample images of faces each corresponding to one of thespecific expression and expressions other than the specific expression,and computes the expression evaluation value based on the magnitude ofprojection components of a vector based on an image signal of the facedetected by the face detector with respect to a vector on thedetermination axis, the determination axis is computed based on datasuch that data of a vector based on the sample images isdimension-compressed by principal component analysis, and the expressionevaluation section normalizes the image signal of the face detected bythe face detector to an image size of the sample image used when thedetermination axis was determined, and computes the expressionevaluation value using the image signal after the normalization.
 8. Theimage-capturing apparatus according to claim 7, wherein the expressionevaluation section performs, on the image signal after thenormalization, a masking process for masking a signal in an image regionthat substantially does not exert an influence on expressiondetermination, and computes the expression evaluation value by using theimage signal after the masking process.
 9. The image-capturing apparatusaccording to claim 1, wherein the notification section displays thenotification information corresponding to the expression evaluationvalue, together with an image being captured, on a display apparatus,the display screen of the display apparatus being oriented toward theimage-captured person side.
 10. The image-capturing apparatus accordingto claim 9, wherein the notification section displays, as thenotification information, a bar chart image indicating the magnitude ofthe expression evaluation value by using the length of a bar on thedisplay apparatus.
 11. The image-capturing apparatus according to claim9, wherein the notification section displays, as the notificationinformation, a frame around the face of an image-captured persondisplayed on the display apparatus, and changes at least one of thecolor, the brightness, and the line type of the frame according to theexpression evaluation value.
 12. The image-capturing apparatus accordingto claim 9, wherein the notification section displays, as thenotification information, character information that differs inaccordance with the magnitude of the expression evaluation value on thedisplay apparatus.
 13. The image-capturing apparatus according to claim9, wherein, when the face detector detects a plurality of faces, theexpression evaluation section computes the expression evaluation valuefor each of the detected faces, and the notification section displaysthe notification information corresponding to the expression evaluationvalue for each face in such a manner as to correspond to each of thefaces of the image-captured persons.